{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started\n",
    "\n",
    "### Importing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computational Graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Const:0\", shape=(), dtype=float32) Tensor(\"Const_1:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "node1 = tf.constant(3.0, tf.float32)\n",
    "node2 = tf.constant(4.0)\n",
    "\n",
    "print(node1,node2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.0, 4.0]\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "print(sess.run([node1, node2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we can add the third node which add node1 and node2.\n",
    "<img src=\"https://www.tensorflow.org/images/getting_started_add.png\" />\n",
    "As it stands, this graph is not especially interesting because it always produces a constant result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node3: Tensor(\"Add:0\", shape=(), dtype=float32)\n",
      "Session.run(node3) 7.0\n"
     ]
    }
   ],
   "source": [
    "node3 = tf.add(node1,node2)\n",
    "print(\"Node3:\",node3)\n",
    "print(\"Session.run(node3)\", sess.run(node3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Placeholder: Used for parameterized to accept externam inputs. A placeholder is a promise to provide a value later\n",
    "<img src=\"https://www.tensorflow.org/images/getting_started_adder.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = tf.placeholder(tf.float32)\n",
    "adder_node = a + b # provides a shortcut to tf.add(a,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.5\n",
      "[ 3.  7.]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(adder_node,{a: 3, b: 4.5}))\n",
    "print(sess.run(adder_node, {a: [1,3], b: [2, 4]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also make complex computation graph by using multiplication operation. The preceding computational graph would look as follows in TensorBoard:\n",
    "\n",
    "<img src=\"https://www.tensorflow.org/images/getting_started_triple.png\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22.5\n"
     ]
    }
   ],
   "source": [
    "add_and_triple = adder_node * 3\n",
    "print(sess.run(add_and_triple,{a: 3, b: 4.5}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the model trainable, we need to be able to modify the graph to get new outputs with the same input. Variables allow us to add trainable parameters to a graph. They are constructed with a type and initial value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = tf.Variable([.3], tf.float32)\n",
    "b = tf.Variable([-.3], tf.float32)\n",
    "x = tf.placeholder(tf.float32)\n",
    "linear_model = W * x + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.          0.30000001  0.60000002  0.90000004  1.20000005]\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(linear_model, {x: [1,2,3,4,5]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = tf.placeholder(tf.float32)\n",
    "squared_deltas = tf.square(linear_model - y)\n",
    "loss = tf.reduce_sum(squared_deltas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see EPE is very high. We can manually adjust the value of W and b to reduce the EPE. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array([-0.99999952], dtype=float32), array([ 0.99999833], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "train = optimizer.minimize(loss)\n",
    "\n",
    "sess.run(init)\n",
    "for i in range(1000):\n",
    "    sess.run(train, { x:[1,2,3,4,5], y:[0,-1,-2,-3,-4] })\n",
    "\n",
    "print(sess.run([W,b]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "#train data\n",
    "x_train = [1,3,2,4]\n",
    "y_train = [0,-2,-1,-3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize variables and placeholders.\n",
    "Tensorflow depends on us telling it what it can and cannot modify. Tensorflow will modify the variables during optimization to minimize a loss function. To accomplish this, we feed in data through placeholders. We need to initialize both of these, variables and placeholders with size and type, so that Tensorflow knows what to expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Model Parameters \n",
    "W = tf.Variable([0.3], tf.float32)\n",
    "b = tf.Variable([0.3], tf.float32)\n",
    "\n",
    "#Model input and outputs \n",
    "x = tf.placeholder(tf.float32)\n",
    "y = tf.placeholder(tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the model structure.\n",
    "After we have the data, and initialized our variables and placeholders, we have to define the model. This is done by building a computational graph. We tell Tensorflow what operations must be done on the variables and placeholders to arrive at our model predictions. We talk more in depth about computational graphs in chapter two, section one of this book.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Model\n",
    "linear_model = W * x + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Declare the loss functions.\n",
    "After defining the model, we must be able to evaluate the output. This is where we declare the loss function. The loss function is very important as it tells us how far off our predictions are from the actual values. The different types of loss functions are explored in greater detail in chapter two, section five."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Loss\n",
    "loss = tf.reduce_sum(tf.square(linear_model - y))\n",
    "\n",
    "#optimizer\n",
    "optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "train = optimizer.minimize(loss)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize and train the model.\n",
    "Now that we have everything in place, we create an instance or our graph and feed in the data through the placeholders and let Tensorflow change the variables to better predict our training data. Here is one way to initialize the computational graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W: [-0.99999791] b: [ 0.99999392] Loss: 2.52847e-11\n"
     ]
    }
   ],
   "source": [
    "#initialization\n",
    "init = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(init)\n",
    "\n",
    "# training loop\n",
    "for i in range(1000):\n",
    "    sess.run(train, {x: x_train, y: y_train})\n",
    "    \n",
    "#evaluate training accuracy\n",
    "curr_W, curr_b, curr_loss =  sess.run([W,b, loss], {x: x_train, y: y_train})\n",
    "print(\"W: %s b: %s Loss: %s\" %(curr_W, curr_b, curr_loss))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tf.contrib.learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (parser.py, line 158)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"/home/amanullahtariq/anaconda3/lib/python3.5/site-packages/dateutil/parser.py\"\u001b[0;36m, line \u001b[0;32m158\u001b[0m\n\u001b[0;31m    l.append(\"%s=%s\" % (attr, `value`))\u001b[0m\n\u001b[0m                              ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "# NumPy is often used to load, manipulate and preprocess data.\n",
    "import numpy as np\n",
    "\n",
    "# Declare list of features. We only have one real-valued feature. There are many\n",
    "# other types of columns that are more complicated and useful.\n",
    "features = [tf.contrib.layers.real_valued_column(\"x\", dimension=1)]\n",
    "\n",
    "# An estimator is the front end to invoke training (fitting) and evaluation\n",
    "# (inference). There are many predefined types like linear regression,\n",
    "# logistic regression, linear classification, logistic classification, and\n",
    "# many neural network classifiers and regressors. The following code\n",
    "# provides an estimator that does linear regression.\n",
    "estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)\n",
    "\n",
    "# TensorFlow provides many helper methods to read and set up data sets.\n",
    "# Here we use `numpy_input_fn`. We have to tell the function how many batches\n",
    "# of data (num_epochs) we want and how big each batch should be.\n",
    "x = np.array([1., 2., 3., 4.])\n",
    "y = np.array([0., -1., -2., -3.])\n",
    "input_fn = tf.contrib.learn.io.numpy_input_fn({\"x\":x}, y, batch_size=4,num_epochs=1000)\n",
    "#input_fn = tf.contrib.learn.io.numpy_input_fn({\"x\": x}, y, 4, num_epochs=1000)\n",
    "\n",
    "# We can invoke 1000 training steps by invoking the `fit` method and passing the\n",
    "# training data set.\n",
    "estimator.fit(input_fn=input_fn, steps=1000)\n",
    "\n",
    "# Here we evaluate how well our model did. In a real example, we would want\n",
    "# to use a separate validation and testing data set to avoid overfitting.\n",
    "print(estimator.evaluate(input_fn=input_fn))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A custom model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "def model(features, labels, mode):\n",
    "    # Build a linear model and predict values\n",
    "    W = tf.get_variable(\"W\", [1], dtype=tf.float64)\n",
    "    b = tf.get_variable(\"b\", [1], dtype=tf.float64)\n",
    "    y = W*features['x'] + b\n",
    "    # Loss sub-graph\n",
    "    loss = tf.reduce_sum(tf.square(y - labels))\n",
    "    # Training sub-graph\n",
    "    global_step = tf.train.get_global_step()\n",
    "    optimizer = tf.train.GradientDescentOptimizer(0.01)\n",
    "    train = tf.group(optimizer.minimize(loss),tf.assign_add(global_step, 1))\n",
    "    # ModelFnOps connects subgraphs we built to the\n",
    "    # appropriate functionality.\n",
    "    return tf.contrib.learn.ModelFnOps(mode=mode, predictions=y,\n",
    "      loss=loss,\n",
    "      train_op=train)\n",
    "    \n",
    "    \n",
    "\n",
    "estimator = tf.contrib.learn.Estimator(model_fn=model)\n",
    "\n",
    "#def data set \n",
    "x = np.array([1.0,2.,3.,4.])\n",
    "y = np.array([0.,1.,2.,3.])\n",
    "\n",
    "input_fn = tf.contrib.learn.io.numpy_input_fn({\"x\": x}, y, 4, num_epochs=1000)\n",
    "\n",
    "#train\n",
    "estimator.fit(input_fn=input_fn, steps=1000)\n",
    "\n",
    "# evaluate our model\n",
    "print(estimator.evaluate(input_fn=input_fn, steps=10))\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
